DOCUMENT RESUME 



ED 369 230 



EC 302 960 



AUTHOR 
TITLE 



INSTITUTION 



SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 

AVAILABLE FROM 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



Ysseldyke, James; And Others 

Implementation of Alternative Methods for Making 
Educational Accountability Decisions for Students 
with Disabilities. Synthesis Report 12. 
National Association of State Directors of Special 
Education, Alexandria, VA. ; National Center on 
Educational Outcomes , Minneapol is , MN. ; Saint Cloud 
State Univ. , MN. 

Special Education Programs (ED/OSERS) , Washington, 
DC. 

Mar 94 

H159C00004 

32p. 

Publications Office, National Center on Educational 
Outcomes, University of Minnesota, 350 Elliott Hall, 
75 East River Rd. , Minneapolis, MN 55455 ($10). 
Reports - Descriptive (141) — Viewpoints 
(Opinion/Position Papers, Essays , etc.) (120) 

MF01/PC02 Plus Postage. 
'''Accountability; *Data Collection; Data 
Interpretation; ^Decision Making; *Disabi 1 i ties ; 
Educational Objectives; Educational Policy; 
Educational Practices; Elementary Secondary 
Education; Evaluation Methods; Government Role; 
Individualized Education Programs; Information Needs; 
'^Outcomes of Education; ^Program Effectiveness; 
Program Evaluation; State Departments of Education; 
Student Evaluation; Student Placement 



ABSTRACT 

This paper examines issues in gathering data for 
purposes of making accountability decisions for students with 
disabilities. The first section describes five alternative methods 
being used: (1) administration of norm-referenced tests; (2) focus on 
results or outcomes; (3) Individualized Education Program 
aggregation; (4) secondary analysis of extant data; and (5) 
establishment of an accreditation program. The second section looks 
at the student population and accountability decision-making 
practices in representative states (Illinois, Iowa, Kentucky, 
Maryland, Michigan, and West Virginia). Considered in the third 
section are the kinds of issues State Education Agency (SEA) 
personnel encounter and the barriers they run into as they attempt to 
document educational results — for students in general and especially 
for those with, disabilities. These include agreeing on outcomes and 
standards, inclusion of students with disabilities in assessments, 
deciding on how and what to report, determining the role of the 
state, deciding on sanctions and rewards, and overcoming fear of 
accountability. The fourth section identifies the kinds of data SEA 
and national education policy personnel feel are needed to 
demonstrate that education works for students with disabilities. 
Section 5 describes ways in which SEA personnel have been successful 
in overcoming barriers. The paper closes by recommending promising 
practices for making accountability decisions. (Contains 10 
references . ) (DB) 
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Implementation of Alternative Methods for Making Educational Accountability 
Decisions for Students with Disabilities 



During the past 10 years there has been a major move to reform America's schools. The 
move has taken on many variations, but in general each of these has been in response to a concern: 
America's students are not doing as well in school as are students in other nations, and they are not 
doing as well as they could. At least a portion of school personnel are always engaged in some 
form of reform, but efforts to reform schools increased significantly following the publication in 
1983 of A Nation at Risk (National Commission on Excellence in Education, 1983). Over the 
decade since publication of that report there have been major actions proposed, the most obvious of 
which were President Bush's America 2000 and now President Clinton's proposed Goals 2000. 
Explicit in both of these pieces of legislation is the notion that America's schools must get better, 
that we must hold America's students to higher standards, and that we can have both excellence 
and equity (all of America's students arc to have an opportunity to learn the same high content 
standards). But, how would we know if this happened? 

One of the characteristics most apparent in the school reforms now taking place across the 
nation is a shift from a focus on documenting the process of educating students to a focus on the 
outcomes of the educational process. Policymakers, legislators, school administrators, and the 
general public want to know the extent to which education in America is working. At each level of 
governance educators are striving to document educational effectiveness, and many different kinds 
of accountability practices are being used Accountability typically is defined as "a systemic method 
to assure those inside and outside the educational system that schools are moving in desired 
directions (Westat, in press). 

The major way of accounting for educational results for America's students has been large- 
scale testing. Large-scale tests have been used by national agencies (e.g., the tests used as part of 
the Nation J Assessment of Educational Progress), by state education agencies (either state 
constructed tests or commercially available norm-referenced measures), and by local education 
agencies. Individual states have been experimenting with alternative methodologies for 
demonstrating accountability for individuals and programs (ETS, 1993; Shriner, Spande, & 
Thurlow, 1994). Among these alternatives are ones that come under the broad heading "authentic 
assessment." Some states have been trying to use performance assessments, portfolios, and 
ouicomes-based assessments for purposes of accountability. 

In this paper we examine some of the issues and challenges in gathering data for purposes 
of making accountability decisions for students with disabilities. In the first section we describe 
the alternative methods that are being used, and in the second section we describe the student 
population and accountability decision-making practices in representative states. We then shift in 
the third section to describe the kinds of issues State Education Agency (SEA) personnel encounter 
and the barriers they run into as they attempt to document educational results - for students in 
general and especially for those who have disabilities. In the fourth section of the paper we 
identify the kinds of data SEA and national education policy personnel say would be needed to 
demonstrate that education works for students with disabilities. 

Section five of the paper is devoted to a description of ways in which SEA personnel have 
been successful in overcoming barriers. We close the paper by recommending promising practices 
for making accountability decisions. 

The points made in this paper are a result of our reading about and hearing descriptions of 
state practices. The latter was in the form of a working seminar jointly sponsored by the National 
Center on Educational Outcomes (NCEO) and the National Association of State Directors of 
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Special Education (NASDSE) in Chicago in September, 1993. The topic of the seminar was 
"Accountability for the Education of All Chi iren." We convened State Directors of Special 
Education and State Assessment Coordinators from each of six states: Illinois, Iowa, Kentucky, 
Maryland, Michigan, and West Virginia. Some of the information about barriers and ways to 
overcome them was drawn also from attendance at a special Forum on Educational Evaluation 
sponsored by the National Association of State Directors of Special Education in Alexandria, 
Virginia in August, 1993. 

Alternative Ways to Demonstrate that Education Works for 
Students with Disabilities 

The primary way in which national, state, and local educators currently make accountability 
decisions for students with disabilities is through child count and compliance monitoring. Local 
Education Agencies (LEAs) receive dollars based on numbers of students identified as students 
with disabilities. Sometimes straightforward allocations are made. Some states employ weighted 
pupil funding — allocating different amounts of money based on students' special education 
conditions (e.g., mental retardation, autism, learning disability) or the severity of the disability. 
Funding may be removed as a consequence of the compliance monitoring process, an investigation 
of the extent to which LEA personnel are complying with state or national policy on delivery of 
services. LEAs and states must report the procedures they use to identify, certify, and deliver 
services to students, and national or state monitors arrive on the local scene to make judgments 
about whether state or national laws have been followed 

Monitoring of special education programs is based on the assumption that application of 
optimal inputs and processes will inevitably lead to positive outcomes for children. Yet, 
documenting the prese: ce of a student in a program (child count), or the compliance of a program 
with regulations, does not tell us whether students are achieving the desired results. 

We identified five alternative ways in which to demonstrate that education works for 
students with disabilities. 

Administration of norm-referenced tests 
Focus cn results or outcomes 
IEP aggregation 

Secondary analysis of extant data 
Accreditation program 

These are described in this section of the paper. 

Administration of Norm-Referenced Tests 

Norm-referenced assessment has remained an integral part of most state accountability 
systems. Some states have a long history of constructing their own state assessments and 
collecting and reporting data on pupil performance on their tests. Others use commercially 
available devices (such as the Stanford Achievement Test or Iowa Tests of Basic Skills) to make 
accountability judgments. And, of course, states make judgments about the relative performance 
of their students compared to students in other states based on NAEP results. Most students with 
disabilities are excluded from participation in state and national testing programs (McGrew, 
Thurlow, Spiegel, & Shriner, 1992). 



6 



Synthesis Report 12 



Focus on Results or Outcomes 

Many of the alternative accountability practices emphasize educational results rather than 
inputs and processes. The educational outcomes of all students may be measured against a set of 
standards (a derivative of a criterion-referenced approach). For example, Illinois expects the 
average performance of a school's students to be at or above a cut-score on their state assessment. 
Schools that fail to meet that standard run the risk of being placed on an "academic watch" list and 
possibly dismantled. Desired educational outcomes are specified, and outcomes may be measured 
by a number of indicators. 

Other states have specified desired educational outcomes and use either norm-referenced or 
performance tests to make judgments about whether students are achieving the outcomes. When 
states engage in performance assessments, varying kinds of performance measures are being used 
The most common form of performance tests are those that require construction rather than 
selection of a response (Coutinho & Malouf, 1993). Unlike most multiple choice exams, 
performance assessments are conducive to developing and assessing higher order skills such as 
creativity, problem solving, and cooperation. The most common performance assessments are 
writing samples. 

Increasingly states are engaging in portfolio assessment in an effort to demonstrate that 
students are meeting outcomes or standards. Kentucky is currently using portfolio assessments as 
a third component of the Kentucky Instructional Results Information System (K3RIS). Writing 
and mathematics products are collected in student portfolios during each "accountability grade" (4, 
8, and 12). Every student is expected to participate by submitting the best examples of their work. 
The Kentucky portfolios are expected to drive the future development of the state's accountability 
system. Representatives from the Maryland Department of Education reported that portfolio 
assessment also is a major approach being considered, especially at the high school level. 
According to Barton and Coley (1994), at least 30 states now have some form of nontradtional test 
items in their statewide assessment - of these, nine states list portfolios. 

IEP Aggregation 

School districts are required by law to have on file an individual educational program (IEP) 
for each student with a disability. In the IEP, school personnel are to specify instructional 
objectives for the student and ways in which the objectives will be measured. Policymakers have 
begun to examine the possibility of collecting accountability data from lEPs. A data-base may be 
constructed to document the attainment of IEP goals or objectives. It is thought that accountability 
might be demonstrated by aggregating data on accomplishment of objectives. State personnel talk 
about the possibility of reporting the numbers of students who have accomplished specific 
objectives, or categories of objectives. 1 

Secondary Analysis of E xtant Data 

Another way to demonstrate accountability would be to re-analyze extant data (data that 
already exist). Secondary data analyses may provide additional accountability information on the 
educational outcomes of students with disabilities. In some states, an extant data analysis may 
even allow student outcomes data to be disaggregated by type of disability. The viability of this 
approach necessarily depends on the existence and quality of previously-collected data (McGrew, 
Spiegel, Thurlow, Ysseldyke, Bruininks, Shriner, 1992). 

Accreditation Program 

A proposal made by some seminar participants for demonstrating accountability for 
educational outcomes of students with disabilities was to establish an accreditation program. A 
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few states, including Maryland and West Virginia, accredit schools for producing positive results 
in student achievement, low dropout rates and high attendence. An accreditation system may be 
used as a means for reporting student performance and distributing rewards and sanctions to 
schools and school districts. Professional organizations have successfully used similar methods 
for securing a high level of performance from both programs and students. Iowa currently 
accredits schools that demonstrate compliance with program standards during a three year onsite 
evaluation and district annual desk audits.. 

Student Population and Accountability Practices in 
Representative States 

Information on the six states that participated in the seminar reflects the range of practices in 
states. The extent to which students with disabilities participate in and are identified in state 
accountability programs is summarized in Table 1 . In Table 2, we provide a summary of the 
general characteristics of the accountability practices in each state. The following paragraphs 
describe the states in greater detail. 

Illinois 

Illinois is a state with a total school age population approaching 2 million students; 12. 1 
percent of the students in Illinois received special education services in 1990-91. Within the past 
few years, Illinois has increased its emphasis on state-level accountability for student achievement 
State law in 1991 established standards, school compliance criteria, and local performance and 
improvement guidelines (the latter are measured by local assessments). The major state assessment 
tool, the Illinois Goal Assessment Program (IGAP), was changed in 1991 from a comparative 
assessment to a criterion-referenced assessment; it covers six basic learning areas. 

Students with disabilities are required by a 1992 law to be included in the IGAP if they can 
take the test, or any component of it, with or without accommodations. Rules now are being 
prepared for determining which students will participate in IGAP, and with what accommodations. 
In addition to IGAP participation, students must participate in the local assessment or alternatives 
to the local assessment. At the same time, the effectiveness of EPs are being examined in Illinois. 

Iowa 

Iowa is a state with a general education population of one-half million students; 1 1.2 N 
percent of the students in Iowa received special education services in 1990-91 . An existing data 
collection system was used as the basis for the current Iowa accountability program. This system 
includes information on initiatives for change, progress in addressing the national education goals, 
enrollment, finance, staff, program, and pupil outcomes. The system includes a wide range of 
measures, from standardized tests (Iowa Test of Basic Skills, Advanced Placement Programs) to 
various participation rates (e.g., dropout rates, attendance rates, participation in Advanced 
Placement courses). In addition to these data, Iowa collects information on program and financial 
accountability. 

Students with disabilities have not, to date, been the focus of systematic efforts to include 
them in the regular data collection programs. Very little information is even available on whether 
students with disabilities are included in the pupil outcome measures. Data on participation (e.g., 
attendance, dropout rates, pursuing postsecondary training) may include students with disabilities. 

Kentucky 

Kentucky is a state with a general education population of more than one-half million 
students; 1 1 .4 percent of the students in Kentucky received special education services in 1 990-9 1 . 
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Table 2 



State Accountability Practices 





Illinois 


Iowa 


Kentucky 


Maryland 


Michigan 


West 
Virginia 


Student Accountability 








X 


X 




Program Accountability 


X 


X 


X 


X 


X 


X 


State Outcomes, Standards 
and/or Indicators targeted 


X 




X 


X 


X 


X 


Data based on other: student 
present, participation, 
demographic; teacher 
characteristics 


X 


X 


X 


X 


X 


X 


Data based on individual test 
performance 




X 


X 


X 


X 


X 


State guidelines for 
accommodations 


0 




X 




X 


X 


Rewards and sanctions 
(accreditation) 


X 


X 


X 


X 


X 


X 


Report data to constituents 




X 


X 


X 


X 




State database established 






X 


X 


X 




Inclusion in assessment 


DSP 




X 


B5P 


X 


mp 



X Implemented 

O Underdevelopment - 

IEP decision is made by IEP team and included on the student's IEP. 
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As part of the Kentucky Education Reform Act (KERA) of 1990, Kentucky instituted a high stakes 
accountability system known as the Kentucky Instructional Results Information System (KIRIS). 
KERIS measures progress toward six learning goals and seventy-five academic expectations, as 
well as assessing non-cognitive indicators such as attendance, dropout and retention rates, and 
proportion of students making successful transitions to work, postsecondary education, or the 
military. At this time, physical and mental health barriers to learning are not in the accountability 
indices. The inclusion of these is being reviewed Learner outcomes are measured at grades 4, 8, 
and 12 in three ways: (a) a transitional assessment, which includes both multiple choice and open- 
response items, (b) performance events, which involve students working cooperatively and 
individually to solve real-life problems through simulations, and (c) portfolios, which are 
developed throughout the school year (currently in math and writing). Accountability for results is 
focused at the school level, with rewards (monetary) and the sanctions (ranging) from Level I, 
which requires the development of a school improvement plan, to Level HI, which includes 
declaring a school to be "in crisis" and provided with technical assistance, and possible firing of 
administrators or staff dependent on changes over time in student achievement. 

All students with disabilities are in KIRIS, but may participate in different ways depending 
on the severity of their disability. One way is for the student to participate in the three assessment 
components just as any other student would A second way is for the student to participate in all 
three components with adaptations and modifications that are consistent with the instructional 
strategies specified in the student's IEP or 504 Ran and available to the student during classroom 
instruction. Assistive technology devices are considered to be a form of adaptations or 
modifications for the assessment A third way is for the student to participate only in an Alternate 
Portfolio Assessment The students participating in the third way are those with moderate to 
severe cognitive disabilities (not to exceed 2% of the population without monitoring). The third 
approach was first used during 1992-93, with scoring to be completed during Fall, 1993. The 
scores of students with disabilities, regardless of how they participate, are included in the data 
reported for schools and used in the accountability index. Data reported for schools can be 
disaggregated for students with disabilities by disability category. 

Maryland 

Maryland is a state with a general education population of three-quarter million students; 
1 1.3 percent of the students in Maryland received special education services in 1990-91. Maryland 
holds students accountable for educational outcomes by requiring students to pass a functional test 
of reading, mathematics, writing, and citizenship in order to graduate. The Maryland School 
Performance Program was implemented in 1991 to hold schools accountable for student 
performance (including achievement, attainment, and participation) and reaching specified 
standards. These data, along with supporting information (student population characteristics, first 
graders with kindergarten experience, students receiving special programs or services, financial 
wealth, staffing ratios, instructional time, nationally norm-referenced achievement data), are 
published annually for each county and the state as a whole. Maryland has implemented the 
Maryland School Performance Assessment Program (MSPAP) to measure higher order thinking at 
grades 3, 5, and 8. Standards for these assessments have been adopted and added to the Maryland 
School Performance Report. Schools will have to reach the designated standards by 1996. In the 
report, school improvement notes that were developed in response to the data also are documented. 

Students with disabilities are included in the data presented in the Maryland School 
Performance Report, both in the student performance and the supporting information sections. 
The data for students with disabilities are not presented separately in the report. Each report also 
documents the numbers of students with disabilities who were exempted from the assessments. It 
is expected that these numbers will be consistent across counties, and that the numbers will reflect 
relatively small percentages of the student population. Parallel special education reports are also 
produced. 
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Michigan 

Michigan is a state with a general education population of one and one-half million 
students; 9.5 percent of the students in Michigan received special education services in 1990-91. 
Accountability in Michigan is based on the Michigan Educational Assessment Program (MEAP) 
and the 1990 Public Act 25. Although MEAP started in 1969, it has been modified over time to the 
point where it now collects information on student performance in reading, math, and science on a 
regular basis, and periodically in social studies and health (with a writing assessment in 
development). MEAP has become more high stakes through recent legislation (such as Public Act 
25, which requires school districts to submit annual reports to constituents and data disks to the 
state department of education). In addition, students must meet established levels of performance 
to receive a state endorsement on their high school diplomas (and eventually, graduation itself may 
depend on performance on MEAP). Three additional statewide accountability mechanisms 
currently are being developed in Michigan. One is the development of assessments specifically to 
assess student proficiency on the Michigan Model Cote Curriculum, which was part of Public Act 
25. Also in Act 25 is the requirement that districts develop their own core curriculum for all 
students, participate in Michigan's accreditation program, and submit three-to-five year school 
improvement plan. The third accountability mechanism is the Governor's Report Card, which 
will document performance on the core curriculum assessments, as well as information on districts 
(such as personnel, staff ratios, revenues, expenditures per pupil, etc.). 

Students with disabilities historically did not participate fully in MEAP, and districts did not 
consistently include students with disabilities in reports provided to the state. Although the state 
developed guidelines in 1992 for including students with disabilities in MEAP, the guidelines were 
inconsistently implemented across the state. The only accommodation that was available for 
MEAP was a Braille form of the test As the MEAP has become more high stakes in nature, there 
has been greater interest in increasing the numbers of students with disabilities who participate. 
This, in turn, has gmerated greater focus on making accommodations for students with 
disabilities. The recent requirement that districts submit annual reports and data disks does not 
entail reporting or including students with disabilities, except for specialized schools. These 
schools, such as ones that provide only special education programs, must report the purpose of the 
school, the number of participating students, the procedures used to enroll them, and the extent to 
which goals of the school were met Similarly, for the requirement that districts develop their own 
core curriculum, participate in the accreditation program, and submit improvement plans, districts 
are encouraged (but not mandated) to identify students with disabilities or the programs that serve 
them. The special education section of the state department of education does collect separate data 
on students with disabilities and the programs that serve them (e.g., financial, personnel, student 
count, and follow-up data). Special education in Michigan also has directed considerable resources 
to the identification of desired outcomes for each category of disability, trained school personnel on 
these outcomes and assessing them, and started to collect data on progress toward the outcomes. 

West Virginia 

West Virginia is a state with a general education population of just over one-quarter million 
students- 12.0 percent of the students in West Virginia received special education services in 1990- 
91 A performance based accreditation system was established in West Virginia under education 
reform legislation passed in 1988. It is a dual component system consisting of state established 
annual performance measures and standards for various aspects of the educational system (one area 
of focus being special education, where the focus is on the occurrence of program monitoring 
reviews and corrections of cited problems). The performance measures are used to collect data 
annually on student performance by grade level, and ouVr indicators such as attendance, dropout 
rate, promotion rate, class size, operating expenses per pupil, and others. West Virginia s core 
curriculum of learning outcomes was recently replaced by instructional goals that cover all required 
programs of study and a criterion-referenced testing program designed to assess student attainment 

8 

13 



Synthesis Report 12 



of goals. Additional student achievement measures include a state-county writing assessment for 
grades 8 and 10, and the sample-based National Assessment of Educational Progress. 

Data for students with disabilities are not reported separately for the annual performance 
measures. It is not clear to what extent students with disabilities currently participate in these 
assessments. For the new assessments, the inclusion of students with disabilities is determined on 
an individual basis by an Individualized Education Program Committee (IEPC). 



Issues Encountered in Attempting to Gather Accountability Data 

A number of issues related to gathering accountability data were raised during the 
Accountability for the Education of All Children Seminar. The major issue identified was 
attitude. The attitudes of educators, policymakers, parents, and students influence the 
effectiveness of an accountability system by permeating the resolution of additional kinds of issues 
such as: 

• commitment 

• agreeing on outcomes and standards 

• source of responsibility for student learning 

• inclusion in assessment 

• deciding how to report information and what to report 

• determining the role of the state 

• using sanctions and rewards 

• asking whether it is worth it 

• fear 

• local control 

• curriculum-test alignment 

• lack of agreement on how to measure complex skills 

• students in non-graded classrooms 

• absence of utility 

By "attitudes" we mean the opinions people hold on the value of education and whether it is 
important for students to be successful at it, opinions about responsibility for social ills like lack of 
literacy and lack of "readiness" to enter school, views about the importance of data and data 
collection enterprises, and thoughts about how data should be reported (and whether they are 
anybody's business). Attitudes also influence views on who should be included in state and 
national assessments, and the kinds of data that should be reported on those who participate. The 
attitudes of those most directly involved in establishing an accountability system will be discussed 
in relation to the categories of issues listed above. 

Commitment 

One issue that nearly all states said was a major barrier to putting any accountability system 
in place was getting people committed to its importance. Apparently, there is a view in many 
school districts and states that it simply is not important to account for the results of education. If 
the general public holds this attitude, then schools have a difficult time getting their attention with 
data. If educators hold this attitude, then it is tough to get them serious about data collection. 

Agreeing on Outcomes and Standards 

State personnel recounted two issues in developing outcomes and indicators as a basis for 
making accountability decisions. Recently, there has been concern that considering all students, 
including those with disabilities, within the development of educational outcomes, and holding 
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educators accountable for all students achieving these outcomes, would result in lowered 
standards. These concerns have created a backlash against including all students in one set of 
standards. Critics argue that if we try to base accountability decisions on attainment of standards, 
and hold all students accountable for attaining standards, this will result in minimal competency 
testing. Retaining high expectations for all students without lowering standards remains a 
challenge. 



"The issue is developmental appropriateness. Avoiding a definition of minimums and reaching 
consensus could impede the accountability process. [In Maryland], we have teachers developing 
all the tasks, but they are the best teachers and may not fully recognize what is developmentally 
appropriate for all kids; middle class values can easily creep in to the system." 

- Robert Gabrys, Assistant State Superintendent 
Maryland Department of Education 



A second issue related to setting standards or outcomes is that different states are defining 
standards differently. This could lead to inequity among states, where the quality of a student's 
education depends upon the state in which the student resides. The National Education Goals 
Panel and various professional organizations (e.g., National Council of Teachers of Mathematics) 
have presented models and criteria that probably will drive standards-setting activities at the 
National level. If Goals 2000 passes without much change, states will submit their standards for 
approval to the National Education Standards and Improvement Council (NESIC). Yet, state 
standards cannot be established until outcomes are defined. Defining outcomes for all students has 
met with mixed success. For example, as a result of resistance to the concept of outcomes-based 
education, Iowa recently failed to receive endorsement for its nine proposed outcome domains. 
This was due in large part to resistance from citizens opposed to outcomes-based education. 

The issue of choosing outcomes includes ensuring that the curriculum is aligned with the 
goals and objectives. Therefore, reaching consensus on educational outcomes becomes an 
important and often extremely value-laden task. At this stage of developing an accountability 
system, individual attitudes become a critical issue. For example, educational outcomes that take 
into account mastery within post school domains such as transition to post secondary or vocational 
schooling, job skills, and independent living skills are emphasized by some stakeholders. A 
subgroup of these stakeholders stresses the importance of post-school outcomes within the social- 
emotional domain. The desire is for young adults exiting school to be responsible, independent 
and able to cope with stress and frustration. Yet, the nation is confronted with rising illiteracy rates 
and falling international standing in mathematics and science. The tension between emphasizing 
post-school transition skills and academic outcomes is most notable when examining the division 
between general and special education. 

Source of Responsibility for Student Learning 

In the past, the division between regular and special education created separate systems, 
each accountable for its own results. In fact, in some states it was thought that compliance 
monitoring is the accountability system for special education, while test performance was the 
accountability system for regular education. 

School systems now are undergoing restructuring. One way in which many systems are 
doing so is by blurring the distinctions between regular and special education. As students with 
disabilities spend increasing time in general education settings, issues arise regarding responsibility 
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for what they learn. Special educators question the appropriateness of being held totally 
responsible for students who may spend only 30 minutes or less each day with a special educator. 

Increasingly the regular education system is being asked to take responsibility for the 
results these students attain. Yet, there is no accountability system to reinforce this responsibility. 
In addition, there is in some instances an attitude that it is unreasonable and unfair to hold regular 
education or regular educators responsible for the outcomes achieved by students with disabilities. 

Inclusion of Students with Disabilities in Assessments and Adapting Measures to Enable them to 
Participate 

Students with disabilities are under-represented in nearly all states' assessment systems. 
There are few if any incentives for increasing inclusion of students with disabilities in assessments. 
This practice may lead to sorting students and ranking some students as more valuable than others. 



"We can show that weYe including more students with disabilities in the regular classrooms, and 
that they are coming [to school] more and behaving better, but [we have] no data on . . .their 
academic gains because we can't show that they are included in the performance exams." 

- Nancy Johnson, Director 
Office of Special Education 
West Virginia Department of Education 

"We don't exclude [students from assessments] except under [the] IE?. But in many instances, 
children are excluded unreportedly." 

~ James Snyder, Assistant Division Chief 
Division of Instructional Services 
West Virginia Department of Education 



As states move to implement new ways of making accountability decisions, students with 
disabilities present a significant obstacle. It is tough to implement new forms of accountability 
practice. It is much tougher when all students must be included. 

While most states have formal guidelines on who must be included in state and district 
assessments that are conducted for accountability or evaluation purposes (see Figure 1), the criteria 
in the guidelines are generally vague. So, it is little wonder that there is incredible variability in 
who is included and in the numbers of students included in assessments. This then leads to 
another issue, one of making fair comparisons. If different states, districts, or schools exclude 
different numbers or types of students, then unfair comparisons may be made among them. 

The complementary issue to inclusion in assessment is adaptation of measures to 
accommodate students with disabilities. What kinds of accommodations are going to be allowed? 
To what extent do accommodations negatively affect technical adequacy? If Braille and large print 
versions are allowed, should an oral presentation of the test be administered? Many argue that the 
educational outcomes being assessed should dictate the appropriateness of an accommodation. In 
our discussions, some argued that curriculum should dictate accommodations. It is common 
practice in some states to allow students with disabilities to use accommodations that are specified 
in their EEP and used in daily instruction. Another issue then arises. If students with IEPs are 
allowed accommodations, why are not accommodations allowed for students who receive special 
instruction in Chapter I but not formally identified as having disabilities? 
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How and What to Report 

There are major differences of opinion on whether and how the test results of students with 
disabilities should be reported First, it is very difficult to identify students with disables in 
many state data sets. Some states report data on students without indicating the numbers of 
2nts with disabilities who participated in the assessment CKhers report data on student ; widi 
disabmn^separately. If data are reported on students with disabilities, there is disagreement on 
wtether AeSSsdiouW be sorted by disability category. Some states gather and report data 
for separate categories of students, others simply report data on students with disabilities. 

Determining the Role of the State 

The data that the states determine to be necessary for establishing accountability will have a 
large effect on the data collection programs in local schools and districts. This is the basis for the 
fifth issue surrounding accountability practices, that of determining the role t c£ the state. The 
infrastructure of public education needs to be evaluated so that the responsibility ol state 

down^pproach in which SEA personnel decide the kinds of data that will be ******* from 
whom they will be collected. In other states local districts decide what they will collect; this makes 
aggregation efforts more difficult 

Sanctions and Rewards 

In addition to developing the accountability program, state government is often responsible 
for distributing sanctions and rewards. A number of issues arise when determining consequences 
for p^Sfvorable accountability results. The first is identifying the level of accountability. A. 
number of options exist, including holding districts, schools, teachers or students accoujuable for 
student outcomes. The most frequently debated is whether individual students should be held 
acSteS meir educational outcomes. Some argue that student accountability programs such 
as requiring a student to meet a standard level of performance on a state assessment in order to 
SSmproves students' motivation to team. However, omers suggest that h^g^ents 
Sponsible for their educational outcomes is unfair to students with disabilities and students from 
disadvantaged homes. 

Those who argue against policies that make graduation contingent upon passing an exam 
contend that there are other ways to demonstrate a student's value to society than a test score 
Alternative assessment procedures may provide a more ecologically vahd i oie^ure of f°f** 
abilities. However, the* are recent additions to state accountabdity and musuumve a 3 > to 5 year 
trial period before results will be used to distribute sanctions and rewards. The importance of 
resoMng the issue of level of accountability will depend upon the seventy of the sanctions and 
rewards For example, withholding a student's diploma on the basis of poor resuteonffl 
accountability measure is less punitive than not allowing the student to graduate. Similarly high 
sSSTht the form of increased teacher wages have produced many J^^SSiSSS^ 
including corruption in the assessments. Some examples of corruption include maeased rates of 
referral to special education, teachers stealing the test answers, and teachers or admmstraters 
encouraginglow performing students to remain at home during testing days. These examples 
suggSaf hij£ Stakes accountability systems that are based on a single measure cieate situations 
Lt are not favorable to including students with disabilities in the accountability system. 

A final issue related to determining the sanctions and rewards component of an 
accountability program is whether effective interventions automatically result from j^ve 
metres Heh sSkes do not guarantee that educators will know how to remedy the problem of 
US educaSSl o^on^s Systems that employ punitive measures are making the assumption 
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that school personnel know how to achieve positive outcomes for students, but that there is no 
motivation for putting those practices into place. 

Is It Worth It? 

The unintended effects of sanctions and rewards based on accountability data is only one 
variable among many to be assessed. 



"We need to define why we're building an accountability system. Is the accountability effort worth 
all the resources unless school improvement/learning is the result?" 

- Robert Gabrys, Assistant State Superintendent 
Maryland Department of Education 



The cost to local schools and districts in money and personnel time must be evaluated so 
that stakeholders are assured that resources are not being shifted away from direct services to fund 
the accountability program. The benefits of an accountability program must be demonstrated in 
order to retain continued support An accountability program will be most effective if a system is 
established within the program to continuously monitor for unintended effects and to document that 
student results lead to instructional interventions and improved programs. This would require that 
the accountability program improves student performance rather than simply measures and 
documents student outcomes. 



"[We need an] assessment of the accountability system itself. Does the system itself over time 
have the right balances? Is this the right accountability system?" 

- Richard Steinke, State Director 
Division of Special Education 
Maryland Department of Education 



Fear 



One of the greatest barriers to establishing an accountability program is fear of 
accountability. This fear is derived from concerns about local autonomy, inclusion of students 
with disabilities, and loss of resources. 

Mistrust is pervasive in education. Reform has the tendency to result in blaming rather than 
producing collaboration among educators. Mistrust exists not only within education, but also 
within constituent groups that have a vested interest in education, such as legislators, state boaids, 
and advocacy groups. The strife among such groups is notable when the debate focuses on 
including students with disabilities in accountability programs. 

Some educators are concerned that being held accountable for the performance of students 
with disabilities may lower their classroom standing and reduce the opportunity to meet established 
thresholds of success. The resistance to allowing students with disabilities to participate fully in 
the accountability program is supported by previous assessment systems that removed test scores 
of students with disabilities from the reporting process. Parents and advocates of students with 
disabilities may also be wary of an accountabSity system based on outcomes because of the 
potential risk of lost resources for direct services. Many believe that child count and monitoring 
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programs are an effective safeguard to ensure that students with disabilities access services. In 
short, a lack of public acceptance and understanding acts as a major barrier to implementing a more 
inclusive accountability program. 

A third identified cause for fear of a state accountability program was a concern over loss of 
resources to local schools and districts due to collecting state accountability data. State data 
collection methodology is sometimes viewed as cumbersome when compared to the greater 
capacity for more sophisticated and streamlined data at the local and intermediate levels. Educators 
resent collecting data that appear to be ill-defined and fail to be consistently presented in meaningful 
reports. The high cost of collecting and analyzing data is a barrier at many levels of governance. 
The initial expense of streamlining the data collection process, including establishing a 
comprehensive data base and continued local input, must be borne by state funds. The expense of 
collecting and analyzing data will increase for accountability systems that disaggregate data for 
special groups of students. This added expense may reduce an accountability program's ability to 
identify the outcomes of students with disabilities and the effectiveness of the programs that serve 
them* An additional barrier to disaggregating data is that students do not easily fit into one 
category or receive services from only one program. The amount of information collected on child 
characteristics required to resolve the issue of identifying special education students and providing 
the contextual information demanded by many is difficult to determine and very expensive to 
collect 



Most states have a long history of local control over developing and implementing 
education programs. A state accountability program may be viewed as a threat to this local 
autonomy. For example, Iowa State Director Dwight Carlson postulated that state-defined 
outcomes failed to gain the necessary support because state outcomes were viewed as a threat to 
local autonomy. State educational outcomes were feared to be a step toward a state-mandated 
curriculum and it was thought that data collected in these outcome domains would be misused to 
justify closing small rural schools, and advancing other political agendas. A state accountability 
program will constitute a major change in policy for Iowa because it does not even have state- 
defined outcomes or competency testing for teachers and students. In an effort to retain local 
support, particularly from the teacher unions, and local autonomy, state policymakers may hesitate 
to mandate reform policies necessary for an effective accountability program. 

Curriculum-Test Alignment 

The lack of alignment of curriculum and instruction with the accountability program is a 
barrier to implementing an accountability program. A difficulty associated with increasing the 
accountability program's connection to the curriculum and instruction is choosing the factor to 
modify first: outcomes, assessment, or curriculum and instruction. Martha Fields, Executive 
Director of NASDSE, recalled a failed attempt to simultaneously reach consensus on outcomes, 
design innovative curriculum and develop alternative assessments during the time she was in 
Maryland This approach was met with opposition as exemplified by newspapers that asserted that 
time needed to modify all these elements of an accountability program was an attempt to slow 
down implementation. 

Kansas took a different approach but did not meet with better success when they adjusted 
the assessments but not the curriculum to the NCTM standards. A subsequent decrease in test 
scores resulted in an attack on the construe * " dity of the assessment rather than a modification of 
the curriculum. These examples illustrate the tension between a linear approach to implementing an 
accountability program, in which assessments are designed to collect baseline data on student 
performance in each outcome domain, and a more circular approach to implementing an 
accountability program in which there is a focus on instructional validity, outcomes and 
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assessments simultaneously. The latter method is a more bottom-up process and requires 
continuous readjustment of each component toward a united, efficient, outcomes-based 
accountability program. 



"I want to believe that if you have school accountability which identifies clear outcomes, includes « 
authentic assessment such as portfolio assessments and the development of products and 
performances, high teacher expectations and embraces diversity and the belief that all children can 
learn, the instruction and curriculum will create a learning environment which motivates students to 
perform at higher levels. Assessement, curriculum, and instruction must begin to be seen as 
interactive." 

- Nancy LaCount, Branch Manager 
Kentucky Department of Education 
Division of Professional Development 
Instructional Strategies Branch 



Lack of Agreement on How to Measure Complex Skills 

A growing dissatisfaction with traditional assessments has led to increased interest in 
alternative assessments designed to measure higher-order skills. Stakeholders are interested in 
measuring students 1 abilities to solve complex problems, and demonstrate other higher-order 
skills, such as observation, creativity, synthesis and interpersonal skills that are not easily 
measured by traditional assessments. With the increasing popularity of cooperative learning 
curricula and instruction, students are thought to be acquiring greater skills in these domains. Yet, 
there are few data to verify this assumption. By requiring students to construct rather than choose 
from a set of responses, higher-order skills may be assessed as well as students' abilities to 
generalize skills to different, more authentic contexts. However, disagreement arises when a 
specific performance assessment must be chosen. A lack of consensus on a dependent measure 
for demonstrating education's effectiveness in developing complex skills is a barrier to 
implementing an accountability program. A point of contention derives from balancing the 
measures' potential construct validity and its reliability and cost Furthermore, developing 
performance assessments that allow all children to participate requires extensive knowledge of the 
variety of skills mastered by children with disabilities. In addition, new assessments arouse 
anxiety over whether curriculum and instruction are developing the skills assessed by alternative 
assessments. 

Students in Non- graded Classrooms 

One of the barriers seminar participants identified to establishment of accountability 
systems is the fact that many students with disabilities are enrolled in non-graded classrooms. 
Even for students mainstreamed in graded classrooms, there exists conflict over whether 
accommodations provide students with disabilities adequate means to demonstrate their abilities. 
Furthermore, some argue that accommodations, such as extra time and alternative presentations, 
violate the technical adequacy of the assessment Disagreement about which, and in what manner, 
students should be included in the accountability program presents a barrier to be overcome. 

No Utility 

Accountability programs face the possible risk (particularly if they include a lot of 
contextual data) of presenting information that no one understands and that no one uses to improve 
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education. Most educators have an aversion to spending valuable preparation time on managing 
paper work if they do not see the activity as useful. 



Data Needed to Demonstrate that Education Works for 
Students with Disabilities 

Educators are being asked to demonstrate that education is working for students with 
disabilities. A discussion of the kinds of data it would take to demonstrate that education is 
working for these students generated three categories of data: input, process and outcomes. 

Input 

The cost of providing services to students with disabilities is critical to understanding the 
effectiveness of education. Schools collect data on a variety of input variables in order to better 
target limited resources and monitor violations of equity in education. For example, the supporting 
information component of Maryland's accountability program collects input data on resources 
including: money spent per pupil, staffing ratios, and instructional time. Mentation is also 
collected on student characteristics such as the percentage of first graders with kindergarten 
experiences, student mobility among schools, number of students with limited English proficiency, 
and number of students enrolled in Special Education, Chapter 1 and free/reduced meal programs. 
Data on student characteristics are collected in an effort to recognize the variety of student inputs 
that influence the context within which learning occurs. It is hoped that collecting information on 
student characteristics and allocation of the educational system's resources will provide guidance 
for future budgeting, and an index of difficulty of the education task within a specific building. 

Process 

There has been increasing concern that every student is provided with the opportunity to 
learn. So, educators indicate that one piece of information needed is the extent to which students 
with disabilities receive an opportunity to learn. Typically, they refer to exposure to curriculum, 
time spent in school, time spent in regular education settings, or dollars spent on education, as 
aspects of opportunity to learn and, in turn, of the educational process. We will not debate the 
definition of opportunity to learn, since it is the subject of a separate policy direction paper 
(Ysseldyke, Thurlow, & Shin, 1994). Some believe that states should be collecting data on the 
extent to which students with disabilities are integrated into regular education. 

Yet another process variable that educators would like to see measured as one indicator of 
whether education is working for students with disabilities is some index of the expectations for 
performance held by individual students 1 teachers. And, along with this notion, educators suggest 
that we ought to know the extent to vhich IEPs translate into instruction. They indicate that one 
source of information on how well eiucation is working for students with disabilities would be the 
number of students who meet IEP objectives. 

Outcomes 

Seminar participants told us that the kind of data most critical to making decisions about 
whether education is working for students with disabilities is data on outcomes. They said that 
they would believe education was working if students were becoming proficient in academic and 
functional skills, achieving valued social and emotional outcomes, and generalizing what they learn 
in school to everyday life. Seminar participants thought that the long term benefit of special 
education might be documented through longitudinal studies of students 1 successful transition to 
adult life. They thought that data on student transition might be collected from follow-up studies of 
student and parent satisfaction levels, community impact studies including such areas as 
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employment, independent living, community participation, and an analysis from the private sector 
of the extent to which our "product" meets the needs of the labor market. The ultimate goal of 
collecting this type of outcomes-based accountability data is to establish a continuous process loop 
that allows for a comparison between past and current graduates of special education programs. 



"[We need to ask]working relative to what? We don't know what others who have gone through 
the system are doing. We need a comparative." 

- Jeananne Hagen, Assistant Bureau Chief 
Bureau of Special Education 
Iowa Department of Public Instruction 



Overcoming Barriers to Implementing New Forms of 
Accountability Decision Making: Approaches States Have Used 

Seminar participants discussed several alternative approaches that states have taken to 
overcome barriers to accountability systems. Among the ways discussed were: 

staff development 
stakeholder involvement 
funding 

curriculum assessment alignment 
reporting data 
mandating data collection 
using rewards and sanctions 
gaining consensus on outcomes 
building a comprehensive database 
developing guidelines on inclusion and accommodations 
shifting accountability practices 
disaggregating data 
providing incentives 
accrediting local school systems 
learning from other states 

Each of these; is discussed briefly here. 

Staff Development 

Several states have attempted to use staff development to get people committed to the 
collection of outcomes data and to help them understand the importance of gathering accountability 
data. These states contend that one of the greatest sources of fear of accountability systems is a 
lack of technical knowledge about the systems and how to implement thera 

The perceptions of teachers are critical to attaining commitment to an accountability system. 
Staff development at the pre- service level is recommended However, this may require the state to 
allocate portions of its limited resources to teacher training. A staff development program must 
also take into account existing local accountability structures. A program that builds upon local 
structures and offers teachers methods for collecting data useful for instructional purposes is most 
likely to build lasting commitment to a state accountability program. 
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The content of staff development programs should include technical information and 
encourage a shift in paradigm from inputs and processes to outcomes. Educators must have 
technical knowledge of administering and scoring alternative assessments to be able to assist in the 
development of a data collection system. This is a critical issue because staff input is a strong 
determinant of the commitment and sense of ownership that an accountability program receives, 
and a technically sound assessment program is a prerequisite for a successful accountability 
program. However, the goal of establishing meaningful outcomes and indicators of progress 
toward these outcomes for all students should not be overshadowed by technical concerns. A 
balance must be struck between collecting data that are statistically adequate and useful for 
improving the quality of education. The resolution of this issue requires a commitment on the part 
of the technical review committees, teachers and other stakeholders to measuring outcomes rather 
than solely recording input or monitoring process. 

The success Maryland has experienced with performance assessment is due" in part to the 
presence of a teacher training consortium in all 24 school systems. The expense and commitment 
required to provide adequate training on designing, administering and scoring alternative 
assessments were identified as barriers to implementing new forms of accountability. Nancy 
LaCount highlighted the importance of local input into Kentucky teacher training programs. 
Kentucky uses cluster leaders to train locally. Teachers are included in the design of assessment 
tasks and scoring portfolios in Kentucky. This not only facilitates stakeholder buy-in and 
involvement, but also serves as professional development activities. Kentucky also has 
professional development consortiums and a network of assessment coordinators in local school 
districts. The goal of this program is to build larger and larger pathways of awareness and sources 
of support for teachers participating in state performance assessments. The results have created 
multiple-levels of knowledgeable stakeholders of the accountability program. The commitment to 
the program is indicated by teachers' willingness to volunteer their time for training other teachers. 

Teacher training is also critical for encouraging the inclusion of all children in the 
accountability program. In the Michigan schools, personnel participate in various staff 
development activities designed to encourage accountability for all students. At the state level, an 
advocacy approach has been used during meetings of the Michigan Department of Education staff 
to encourage an inclusive approach to accountability. Including representatives of special 
education in the design of alternative assessments and teacher training programs was recommended 
for promoting awareness and ensuring the feasibility of the inclusion of all students in the 
accountability program. The separate regular and special education systems may lead regular 
education teachers to design assessments that are not developmentally appropriate for students with 
disabilities. Special education instructors were involved in designing Maryland performance 
assessments. Additional guidance has been obtained by Maryland and Kentucky from higher 
education programs. 

Get Stakeholder Involvement up Front Often in th e Form of LEA/SEA Task Forces 

Two approaches to reducing the barriers of perceived threat to local autonomy were 
discussed. Each solution addressed the amount of control local educators exert over developing 
and accessing accountability data elements. 

One approach is to use the intermediaries, such as SEAs and LEAs, to design and 
implement state accountability data collection programs. For example, West Virginia requires 
LEAs to provide data that identify an area of concern in order to receive federal discretionary 
funds. A restriction on the type of data acceptable for this purpose requires that the data be in 
accordance with the guidelines of the West Virgina Special Education Evaluation Review System 
(WVSEERS). However, LEAs soon discovered that existing data did not provide the type of 
information needed for documenting the performance of students with disabilities and the programs 
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that serve them. As a result, LEAs were placed in the position of developing methods for 
collecting performance data needed to receive additional funds. 

Similarly, Iowa formed a board of directors, consisting of local AEAs (area educational 
agencies) and state department staff to develop one system of special education data collection for 
their state. Thus, the ownership of the data collection system in Iowa is at the intermediate unit 
level for special education. 

A second approach to reducing the fear that a state accountability program will compromise 
local autonomy is to include input from a wide range of professionals into the development of the 
data collection program. Michigan formed a coalition of representatives from various national, 
state and local groups, including two teachers unions, the curriculum association, and the national 
association of secondary principals, to work together and mediate the development of data 
collection procedures. Including curriculum developers in the design of assessment procedures 
was considered crucial to aligning a state accountability program to the curriculum. 

Similarly, Maryland emphasized the role of teachers in designing assessments. A group of 
teachers played an active part in developing performance assessments. The assessment program 
was implemented in September of 1992 and the teachers met again in December to evaluate and 
improve the assessment program. The end result was a paper disseminated to all of the teachers. 
State director, Robert Gabrys, stated that the extensive teacher input reduced staff fear of 
performance assessment -because the assessment process was no longer viewed as solely state 
controlled. Furthermore, there is evidence, including support from the teachers union, that the 
emphasis upon teacher input will increase long term commitment to perforniance assessments. 
Teachers have responded favorably to designing and administering performance assessments, as 
well as to evaluating the results of student performance on these measures. Additional comments 
by teachers indicate that they do not perceive a significant loss of instruction time due to the 
performance assessment program. 



"More decisions on assessments are made by curriculum people, not just testing people. That 
forced the alignment to occur. Curriculum and assessment people sit in the same room and are 
forced to come to a consensus." 

- Robert Gabrys, Assistant State Superintendent 
for School Performance 
Maryland Department of Education 



Parents, students and advocacy groups also are important in building local commitment to 
an inclusive accountability program. Kentucky has used mass media, including television, to 
increase public awareness and support among stakeholders for their accountability program- 
Michigan has opened its training centers to parents and has published a brochure emphasizing the 
inclusive nature of Michigan educational reform Similarly, every Illinois Goal Assessment 
Program (IGAP) task force includes a parent component that examines the data to be collected and 
recommends solutions to overcoming barriers to collecting these data. 

Fund T E A s for Data Collection 

In some states funding is provided to LEAs to enable and encourage them to gather data on 
student outcomes. While this practice is not widespread, SEA personnel do talk about the 
possibility of witholding small amounts of the funds that are allocated to LEAs and making the 
distribution of those funds contingent on coDection of outcomes data. 
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Align Curriculum 'with Assessment 

SEA personnel recognize that one of the barriers to collection of data on outcomes is the 
lack of alignment between assessment and curriculum. Test developers at many corporations are 
busy matching their tests and test items to state competencies and outcomes. The practice ranges 
from generic matching to very specific cataloging of items corresponding to specific instructional 
objectives. 

Report Data on Student and School Outcomes 

Awareness of the importance of accountability assessments is increased when the results of 
the assessments are published in formal reports or released to the media. The manner in which 
results of accountability assessments are reported may also increase awareness. Maryland released 
the results of its state assessment to the press and emphasized that the goals that have been set are 
goals for the year 2000. Student performance is reported as baseline information on the magnitude 
of student accomplishments relative to the year 2000 goals. 

Similarly, data reporting has been used to overcome barriers to including students with 
disabilities in the accountability program. Maryland's school pf rformance reports include the 
number of students with disabilities who were exempted from each measure. This information is 
intended to encourage advocacy groups to question local inclusion policies. In addition, 
occurences of inordinate numbers of exemptions are investigated by the state. Excluding students 
who do not meet the necessary requirements for an exemption (children for whom state standards 
are determined to not be appropriate) is also discouraged by assigning zeros as the scores for all 
exempted students. 

Mandate Collection of Accountability Data 

In Kentucky, the inclusion of students with disabilities in the accountability program is 
mandated. All students participate in the Kentucky Instructional Results Information System 
(KIRIS) in one of three ways: (a) participate fully in all assessments with no modifications; (b) 
participate in all assessments with accommodations as specified by IEPs; (c) participate in 
Alternative Portfolio Assessments. Students with disabilities who meet the criteria for the Alternate 
Portfolio Assessment are exempt from the transitional multiple-choice, open response assessment, 
performance events, and mathematics and writing portfolios completed by all other students. They 
complete an alternate portfolio that is included in the accountability program, as are any other 
student assessment results. The students who meet the criteria for Alternate Portfolio Assessments 
generally have moderate to severe cognitive disabilities and comprise only 1-2% of the total student 
population. The criteria for participation in the Alternative Portfolio Assessment were established 
by a program advisory of the Disabilities and Diversity committee, which was established to 
review issues and make recommendations for the development, implementation and inclusion of 
students with disabilities in the KIRIS accountability program. The program advisory is 
responsible for communicating policy decisions to local schools and school districts. 

Additional support for including all students in the Kentucky accountability program 
includes peer support for teachers of students with moderate to severe disabilities, attempting to 
align the curriculum and instruction to the Alternate Portfolio System, and expanding the range of 
appropriate transitions identified by the Kentucky Education Reform Act (KERA) to be more 
inclusive of transitions appropriate for students with disabilities. A final method used by Kentucky 
to include students with disabilities in the accountability system was the development of a statute 
that states "schools shall expect a high level of achievement of all students," thus providing the 
legal basis for high standards for all students. These efforts provide support to all levels of 
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accountability and increase the likelihood that the mandated policy of inclusion will become a 
reality. 

Federal Mandate or Rewards and Sanctions 

Participants did not approve of redefining federal special education funding programs to 
allocate resources contingent upon whether students meet state or school-established standards. It 
was thought that programs serving students with severe disabilities might suffer from this type of 
funding policy. Furthermore, it was argued that it is doubtful that the federal government could 
efficiently and effectively regulate the progress of students with disabilities. In addition, it was 
suggested that given the separate structures of federal education agencies, federal involvement in 
accountability programs might perpetuate the division between special and regular education. 
Overall, the federal government is viewed as a macro structure that should be responsible for 
encouraging inclusive accountability programs through legislation (such as Goals 2000 which calls 
for the development of assessment procedures that include all students) and by example (such as 
increasing the inclusion of students with disabilities in national assessment programs such as 
NAEP). Distributing sanctions and rewards based upon student performance was recognized as a 
state responsibility. 



"Making the availability of federal funds contingent on students meeting outcomes would not be 
equitable and very difficult to regulate*' 

- Richard Baldwin, State Director 
Special Education Services 
Michigan Department of Education 



Gain Consensus on Outcomes 

One of the barriers to getting new accountability practices into place is the lack of 
consensus on outcomes and indicators. Participants at the working session thought that they 
would have a better chance of implementing new accountability practices if there was consensus on 
outcomes and indicators. They supported efforts to develop a conceptual model of outcomes and 
indicators and the development of a consistent system for documenting student outcomes. Once 
agreed upon, outcomes and indicators need to be incorporated into the curriculum. Kentucky used 
teachers from across the commonwealth, representing various disciplines, to develop a curriculum 
framework based on Kentucky's learning goals and expectations. Schools arc revising their 
curriculum and instructional strategies to facilitate Kentucky students attaining the goals and 
expectations. 

Build a Comprehensive State Data Base 

In some states there has been a move to develop comprehensive state data bases that include 
contextual information as well as outcomes data. Some states include in their data bases individual 
and local evaluation reports that are synthesized through meta-analysis procedures. When states 
have data from multiple agency evaluations of schools and services, they are able to use those data 
to plan future services ana to influence policy development, budgeting and legislative actions. 

Develop State Guidelines on Inclusion and Accommodation in Testing 

Most states do not mandate the inclusion of students with disabilities in testing because they 
lack the structures to enforce such a policy and the programs to provide the technical support 
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necessary for ensuring success. However, many participants in the seminar on accountability 
identified the development of state guidelines for inclusion and accommodation practices as a 
critical step to overcoming the barriers to inclusion. Of the states in the seminar, some (Maryland, 
Illinois, and West Virginia) leave the decision of inclusion in assessment to IEP teams to determine 
on an individual basis. Others (Kentucky and Michigan) provide specific guidelines delineating 
which students participate in specific components of the accountability program. The latter policy 
is more advantageous for constructing a true representation of student achievement within a state. 
Leaving inclusion decisions to IEP teams increases the likelihood that students with disabilities will 
be inconsistently included in state accountability programs. State guidelines for inclusion will be 
most effective if they are accompanied by state guidelines for accommodations. Currently, of the 
states participating in the seminar Kentucky, Michigan and West Virginia provide guidelines for 
accommodation practices. 

Shift Accountability Practices to Hold Systems Responsible fo r Producing Environments in Which 
gtudents Can and Will Learn 

Often, outcomes accountability programs are put into place without careful consideration of 
who is responsible for attainment of outcomes. Often, the focus in accountability practices is on 
attainment of a set of outcomes. In some states there is discussion of holding districts accountable 
for producing environments in which students can and will achieve desirable outcomes. Goals for 
improved achievement are set and progress toward the goals is monitored. 

Disaggregate Data 

An approach to overcoming the barriers to inclusion of students with disabilities in 
accountability programs is to disaggregate data for students with disabilities (either by category or 
across categories). This practice would allow the effectiveness of special education jrograms to be 
evaluated In Kentucky, data can be disaggregated for studetns with disabilities by disability 
category at grades four, eight and twelve. Data on students receiving special education in 
Maryland could be disaggregated from students not eligible for special education services. All 
information is reported to stakeholders in aggregate format, but disaggregated information is 
available. Ensuring an equitable education requires collecting data on the progress r>f groups of 
students (including special education and disadvantaged students) toward meeting siandards. 

Disaggregating data for students with disabilities may revolve around establishing different 
assessments. For example, Michigan is developing a set of outcome indicators for each of 12 
disability categories. However, the general consensus of those at our seminar supported collecting 
data on the performance of students with disabilities from state accountability programs rather than 
developing separate assessment procedures for different categories. This would resolve the 
problem of artificially placing students into categories when their abilities and performance fall on a 
wide continuum. Furthermore, students with disabilities would be encouraged to work toward the 
same standards of performance as regular education students. This effort would be evident in 
developing IEPs that outline goals for students that are congruent with state outcomes. IEPs may 
also include data collection procedures for assessing student progress toward these goals. This 
information could be collected from IEPs and reported in disaggregated format with the results of 
state assessments. 

Provide Incentives 

Creating incentive programs was identified as one solution to the barriers to implementing 
new forms to accountability. For example, providing more focused technical assistance programs 
may act as an incentive for educators to account for student performance and to include students 
with disabilities in that accounting. Incentives to include all students in a new accountability 
program may be derived from state-distributed sanctions and rewards. The first step to 



23 



Synthesis Report 12 



implementing a high stakes accountability program is to identify constraints and sources of 
resources. 

A system of sanctions and rewards should be sensitive to the availability of resources and 
emphasize continuous progress toward standards. For example, Illinois has teams evaluate the 
appropriateness of the improvement plans developed by local schools. Then, if the school 
improvement plan fails to result in positive student gain, some of the school's staff is replaced. 
Ultimately, the entire existence of the school, as well as the principal, superintendent, and school 
board's jobs may be terminated. 

Accredit Lo cal School Systems 

A more common approach to distributing consequences for school performance on state 
accountability assessments may be to accredit schools for creating an environment in which all 
children progress toward state standards and outcomes. Of the states represented at the seminar, 
Michigan and West Virginia apply similar methods for encouraging participation in the 
accountability program In West Virginia, accreditation is based on unannounced site visits that 
investigate deficiencies in a school's annual report. Accreditation and local reports provide schools 
with public recognition of good work. 

Learn from Other States 

A final method identified in the seminar for overcoming the barriers to implementing a new 
system of accountability is to learn from other states about programs that have been successful. 
Specifically, individuals involved in developing and implementing accountability programs would 
benefit from specific ideas and examples of effective programs in other states, rather than from an 
overview of how accountability systems can be developed. It is hoped that this report, along with 
the recent NCEO survey of state data managers (see Ysseldyke, Vanderwood, & Reschly, 1994) 
will provide this concrete information, as well as inspire others to share their successes with 
accountability. 

Recommended Practices 

There should be little doubt that the collection of accountability data is an important activity. 
The activity is critical to the continued funding and support of services for students with 
disabilities. The activity is important to provision of data used in making policy decisions. And, it 
is clear that child count data, while interesting and important, do not begin to meet the needs of 
those who want to improve instructional programs and educational outcomes for students with 
disabilities. 

It is one thing to recognize the importance of changing practices in the collection of 
accountability data. Yet, people will not do so just because it is a good idea to do so. Careful 
thought must be given to the creation of incentives for people to change the ways in which they 
gather data. Here we have much to learn from the experiences of the state education agency 
personnel who participated in our seminar. 

First, any data collection program put in place must make sense to those who are gathering 
data. And, it probably is not a good idea to invent new data collection requirements. Rather, 
efforts need to be made to involve stakeholders from the outset in deciding precisely the lands of 
data to be collected and the methods to be used. We recommend that an outcomes-based approach 
be used and that the Self-Studv Guide to the Deve lopment of Educational Outcomes and IatotOTS 
(Ysseldyke & Thurlow, 1993) or a similar document be used to help SEA and LEA personnel 
reach agreement on valued outcomes and kinds of data to be collected It will not be possible at the 
outset to gather data on all outcomes, but even a partial system will be better than none. 
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SEA personnel should consider establishing incentives for data collection. The approach 
used in New Jersey could be followed There, SEA personnel hold a competition among districts 
for funds to support data collection. If SEA personnel are serious about data collection, then they 
need to allocate resources to this activity. 

If educational personnel in states are to spend time gathering data on educational outcomes 
for students with disabilities, then they need training in how to do so. Staff development is critical 
to the accountability process. 

Finally, new methods for collecting accountability data will be effective only if the data are 
actually used in the process of making policy decisions and there is evidence that this was done. 
School personnel are not about to become excited about the collection of data that arc put on the 
shelf rather than put to good use. Usually, for people to see that data were put to good use 
requires that a report be developed to showcase the data. 

There is, we think, an overall goal to the standards-setting process. That goal is one of 
developing broad, meaningful outcomes for all children that allow for the variety of challenges 
faced by students with disabilities. One determinant of the reporting procedure will be the 
audience. Data collected from an accountability assessment may be reported to the school's 
teachers, surrounding neighborhood, local district, or state. Each purpose requires different 
information. Teachers benefit the most from accountability reports that are clear and directly 
related to instruction. Students and parents should be able to access pertinent information with 
little difficulty and prior knowledge of statistics. Data reported at each level of governance is likely 
to vary, with data reported to the state consisting of only a portion of all the data collected by local 
schools and school districts. 



25 

31 



Synthesis Report 12 



References 

Barton, P. E., & Coley., R. J. (1994). Testing in Americas schools . Princeton, NJ: Educational 
Testing Service. 

Coutinho, M., & Malouf, D. (1993). Performance assessment and children with disabilities: 
Issues and possibilities. Teaching Exceptional Children , 25 (4), 63-67. 

McGrew, K. S., Spiegel, A. N., Thurlow, M. L., Ysseldyke, J. E., Bruininks, R. H., & 

Shriner, J. G. (1992). Outcomes for children and vouth with disabilities : Secondary analysis 
of national data collection programs (Working Paper 3). Minneapolis, MN: National Center 
on Educational Outcomes. 

McGrew, K. S., Thurlow, M. L., Shriner, J. G., & Spiegel, A. N. (1992). Inclusion of students 
with disabilities in national and state data collect ion programs (Technical Report 2). 
Minneapolis, MN: National Center on Educational Outcomes. 

National Commission on Excellence in Education. (1983). A nation at risk: The imp erative for 
e ducational reform. Washington, DC: US Government Printing Office. 

Shriner, J. G., Spande, G. E., & Thurlow, M. L. (1994). State special education outcomes 1993. 
Minneapolis, MN: National Center on Educational Outcomes. 

WESTAT. (in press). Issues and options in outcomes-based accountability for students with 
disabilities. 

Ysseldyke, J. E., & Thurlow, M. L. (1993). Self-studv guide to the de velopment of educational 
outcomes and indicators . Minneapolis, MN: National Center on Educational Outcomes. 

Ysseldyke, J. E., Thurlow, M. L., & Shin, H. (1994). Opportunitv-to-leara standards. 
Minneapolis, MN: National Center on Educational Outcomes. 

Ysseldyke, J., Vanderwood, M. L., & Reschly, B.E. (1994). Availability of data on school 
completion outcomes and indicators. Minneapolis, MN: National Center on Educational 
Outcomes. 



26 



